Environmental interrupt in a head-mounted display and utilization of non field of view real estate

ABSTRACT

A wearable computing device includes a head-mounted display (HMD) that generates a virtual reality environment. Through the generation and tracking of positional data, a the virtual environment may be interrupted or paused. Upon pausing the environment, a user may access a number of ancillary menus and controls not otherwise available during normal operation of the virtual environment.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation and claims the prioritybenefit of U.S. patent application Ser. No. 14/283,083 filed May 20,2014, which claims the priority benefit of U.S. provisional patentapplication No. 61/931,583 filed Jan. 25, 2014, the disclosures of whichare incorporated herein by reference.

BACKGROUND OF THE INVENTION

Field of Invention

The present invention generally relates to wearable virtual reality (VR)computing devices having a head-mounted display (HMD). Morespecifically, the present invention relates to interrupting operationsin the field of view in the HMD and utilizing non field of view realestate in the HMD.

Description of the Related Art

Wearable VR systems integrate various elements, such as input devices,sensors, detectors, image displays, and wireless communicationcomponents, as well as image and audio processors. By placing an imagedisplay element close to the eyes of a wearer, an artificial image canbe made to overlay the view of the real world or to create anindependent reality all its own. Such image display elements areincorporated into systems also referred to as head-mounted displays(HMDs). Depending upon the size of the display element and the distanceto the eyes of the wearer, artificial images provided on the display mayfill or nearly fill the field of view of the wearer.

VR systems incorporating an HMD are mobile and lightweight, whileallowing for communication and interaction with a virtual environment.Such systems are generally lacking, however, in that they still requireuse of an independent controller for navigation of the virtualenvironment. In this sense, most HMDs are little more than gogglesallowing for entry into a VR environment. There is a need in the art fornavigation and control of a VR environment without introducing anindependent controller device, especially with respect to interruptingoperations of the environment in a natural and non-intrusive manner.There is a further need to best utilize non-field of view “real estate”in that VR environment.

SUMMARY OF THE CLAIMED INVENTION

Embodiments of the present invention include systems and methods forinterrupting a virtual environment in a head-mounted display.Information may be stored regarding at least one control setting thatassociates a function with a change in position of the head-mounteddisplay. The head-mounted display may be calibrated to identify a startposition. Positional data that tracks movement of the head-mounteddisplay may be generated. A current position of the head-mounted displaymay be determined to be indicative of a change from the start positionthat exceeds the change in position of the control setting. Then, thefunction associated with the control setting may be executed, which mayinvolve interrupting the virtual environment in the head-mounted displayby pausing the environment.

A method for interrupting a virtual environment in a head-mounteddisplay is claimed. Such methods may include storing informationregarding at least one control setting that associates a function with achange in position of the head-mounted display, calibrating thehead-mounted display to identify a start position, generating positionaldata that tracks movement of the head-mounted display, determining thata current position of the head-mounted display is indicative of a changefrom the start position that exceeds the change in position of thecontrol setting, and executing the function associated with the controlsetting, wherein the function comprises interrupting the virtualenvironment in the head-mounted display by pausing the environment.

Further embodiments include system for interrupting a virtualenvironment in a head-mounted display. Such systems may include memorythat stores information regarding at least one control setting thatassociates a function with a change in position of the head-mounteddisplay, at least one of a gyroscope, magnetometer, and an accelerometerthat calibrates the head-mounted display, wherein a start position ofthe head-mounted display is identified and generates positional datathat tracks movement of the head-mounted display, a processor thatexecutes instructions stored in memory to determine that a currentposition of the head-mounted display is indicative of a change from thestart position that exceeds the change in position of the controlsetting and to execute the function associated with the control setting,and a head-mounted display including at least one lens to display thevirtual environment where execution of the function interrupts theenvironment by pausing the environment.

Embodiments of the present invention may further include non-transitorycomputer-readable storage media, having embodied thereon a programexecutable by a processor to perform methods interrupting a virtualenvironment in a head-mounted display as described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of an exemplary wearable computingdevice.

FIG. 2A illustrates an HMD that completely immerses a wearer in avirtual reality environment.

FIG. 2B illustrates an HMD that allows for generation of VR informationwhile maintaining perception of the real world.

FIG. 3 illustrates an exemplary implementation of an interrupt in the VRenvironment.

FIG. 4 illustrates a method for implementing an interrupt in the VRenvironment.

FIG. 5 illustrates the use of non-field-of-view real estate to provideinformation ancillary to the VR environment.

DETAILED DESCRIPTION

Embodiments of the present invention include systems and methods forinterrupting a virtual environment in a head-mounted display.Information may be stored regarding at least one control setting thatassociates a function with a change in position of the head-mounteddisplay. The head-mounted display may be calibrated to identify a startposition. Positional data that tracks movement of the head-mounteddisplay may be generated. A current position of the head-mounted displaymay be determined to be indicative of a change from the start positionthat exceeds the change in position of the control setting. Then, thefunction associated with the control setting may be executed, which mayinvolve interrupting the virtual environment in the head-mounted displayby pausing the environment.

FIG. 1 illustrates a block diagram of an exemplary wearable virtualreality system 100. In communication with an external computing device110, wearable virtual reality system 100 may include a USB interface120, wireless communication interface 130, gyroscope 140, accelerometer150, magnetometer 160, data storage 170, processor 180, and head-mounteddisplay (HMD) 200.

Head-mounted display (HMD) 200 allows its wearer to observe real-worldsurroundings, a displayed computer generated image, or a combination ofthe two. HMD 200 may include a see-through display in some embodiments.The wearer of wearable co virtual reality system 100 may be able to lookthrough HMD 200 in such an embodiment and observe a portion of thereal-world environment notwithstanding the presence of the wearablevirtual reality system 100. HMD 200 in a further embodiment may beoperable to display images that are superimposed on the field of view toprovide an “augmented reality” experience. Some of the images displayedby HMD 200 may be superimposed or appear in relation to particularobjects in the field of view. In a still further embodiment, HMD 200 maybe a completely virtual environment whereby the wearer of the wearablevirtual reality system 100 is isolated from any visual contact with thereal world.

The displayed image may include graphics, text, and/or video; audio maybe provided through a corresponding audio device. The images displayedby the HMD may be part of an interactive user interface and includemenus, selection boxes, navigation icons, or other user interfacefeatures that enable the wearer to invoke functions of the wearablecomputing device or otherwise interact with the wearable computingdevice. The form factor of HMD 200 may be that of eyeglasses, goggles, ahelmet, a hat, a visor, a headband, or in some other form that can besupported on or from the head of the wearer.

To display a virtual image to the wearer, the HMD may include an opticalsystem with a light source such as a light-emitting diode (LED) thatilluminates a display panel. The display panel may encompass a liquidcrystal display panel (LCD). The display panel may generate lightpatterns by spatially modulating the light from the light source, and animage former forms a virtual image from the light pattern.Alternatively, the panel may be liquid crystal on silicon (LCOS) wherebya liquid crystal layer may be situated on top of a silicon backplane.

The HMD in an exemplary embodiment includes a 7 inch screen withnon-overlapping stereoscopic 3D images whereby the left eye sees extraarea to the left and the right eye sees extra area to the right. The HMDattempts to mimic normal human vision, which is not 100% overlapping.The field of view in an exemplary embodiment is more than 90 degreeshorizontal (110 degrees diagonal) thereby filling approximately theentire field of view of the view such that the real world may becompletely blocked out to create a strong sense of immersion.

An embodiment may utilize 1280×800 (16:10 aspect ratio) thereby allowingfor an effective of 640×800, 4:5 aspect ratio per eye. In an embodimentthat does not allow for complete overlap between the eyes, the combinedhorizontal resolution is effectively greater than 640. The displayedimage for each eye is pin cushioned thereby generating aspherical-mapped image for each eye.

HMD 200 may communicate with external computing device(s) 110. Externalcomputing device(s) 110 are inclusive of application servers, databases,and other external computing components known in the art, includingstandard hardware computing components such as network and mediainterfaces, non-transitory computer-readable storage (memory), andprocessors for executing instructions or accessing information that maybe stored in memory.

Wearable virtual reality system 100 may in some instances be physicallyconnected to external computing device(s) 110. Such a connection may beimplemented by way of a USB interface 120, which may be used to senddata to and receive data from an external computing device 110 by way ofa USB-compliant cabling. USB interface 120 may also be used to power thewearable virtual reality system 100 thereby potentially negating theneed for an external power supply and any power cabling associated withthe same. In some instances, a further power adapter (not shown) may benecessary to implement power by way of the USB interface 120. It shouldbe understand that reference to USB is exemplary as other types ofinterfaces may be used including but not limited to FireWire, Lightning,as well as other cabled connection standards such as HDMI and DVI.

Wearable virtual reality system 100 of FIG. 1 includes a wirelesscommunication interface 130. Wireless communication interface 130 may beused for wirelessly communicating with external computing device(s) 110.Wireless communication interface 130 may also be used for communicatingwith other wearable computing devices 100. Wireless communicationinterface 130 may utilize any number of wireless communication standardsthat support bi-directional data exchange over a packet-based networksuch as the Internet. Exemplary communication standards include CDMA,GSM/GPRS, 4G cellular, WiMAX, LTE, and 802.11 (WiFi).

Wearable virtual reality system 100 may include one or more ofthree-dimensional axis gyroscopes 140, accelerometers 150, andmagnetometers 160 Gyroscope 140 may be utilized to measure orientationbased on the principles of angular momentum. Accelerometer 150 may beused to detect magnitude and direction of acceleration as a vectorquantity. This result can be used to sense orientation because directionof weight changes, coordinate acceleration correlated to g-force or achange in g-force, and vibration, shock, and falling in a resistivemedium by way of a change in proper acceleration. Magnetometers 160 maybe used to identify disturbances in a magnetic field relative thewearable virtual reality system 100. Magnetometer 160 can assist in theidentification of true north for GPS and compass applications as well asassist with touchless or camera-less gesture input. By utilizing datagenerated from the foregoing, absolute head orientation tracking withoutdrift relative to the earth may be calculated. Latency tracking mayoperate at approximately 1000 Hz to decrease response time and increaseperceived realism. The displays of wearable virtual reality system 100may be adjusted to allow the individual displays to be moved further orcloser to the eyes of the wearer.

Wearable virtual reality system 100 may operate by way of the executionof non-transitory computer readable instructions stored in data storage170, where execution occurs through operation of processor 180. WhileFIG. 1 illustrates data storage 170 and processor 180 as being presentat wearable virtual reality system 100, such elements may be located inexternal computing device(s) 110 or in some instances, with executableoperations distributed between the two. Processor 180 and executableinstructions at data storage 170 may also control various aspects of USBinterface 120, wireless interface 130, gyroscopes 140, accelerometers150, and magnetometers 160.

FIG. 2A illustrates an HMD 200 that completely immerses a wearer in avirtual reality environment. While FIG. 2A is illustrated as immersivegoggles, other form factors are possible and envisioned. The operationof elements in FIG. 2A are the same as those discussed in the context ofFIG. 2B. FIG. 2A includes head-mounted support 210 that allows forwearable virtual reality system 100 (including HMD 200) to be positionedon the head of a wearer. HMD 200 further includes lens displays 220A and220B that may be of LCD or LCOS construction as described above. Lensdisplays 220A and 220B may be an integrated part of wearable virtualreality system 100.

The manufacture of wearable virtual reality system 100 may allow forintegration of components like those illustrated in FIG. 1 and variouscomponent interconnects to be internally integrated. Other componentsmay be situated on the exterior of wearable virtual reality system 100to allow for more ready access or physical connections to externalcomputing device(s) 110. An embodiment of wearable virtual realitysystem 100 may include a microphone to allow for voice communicationwith other individuals utilizing wearable virtual reality system 100 orto allow for certain hands free control of the system 100.

FIG. 2B illustrates an HMD 200 that allows for generation of virtualreality information while maintaining perception of the real world. Suchdual perception is provided for by not completely immersing the wearerwithin the confines of the virtual environment (i.e., the real world canstill be seen and perceived). While HMD 200 of FIG. 2B is illustrated asa simple band other form factors are possible and envisioned. Theoperation of elements on FIG. 2B are the same as those discussed in thecontext of FIG. 2A.

FIG. 3 illustrates an exemplary implementation of an interrupt in the VRenvironment. As illustrated, the user 310 of HMD 200 is looking “downthe line” or “dead center” of the VR environment 320, the center ofwhich is reflected by ray 330. It should be noted that ray 330 ispresented solely for the purpose of assisting with illustration and isnot literally present in the VR environment 320 although it is possiblethat indicia of orientation could be displayed by the HMD 200 withrespect to the virtual environment 320. As reflected by ray 330 and theline-of-sight of the user (340), both may be relatively parallel to oneanother.

Ray 330, while not a necessary illustrated element in the VRenvironment, may be determined from calibrating the HMD 200 when theuser 310 first mounts the same to their head. By utilizing informationgenerated by one or more of three-dimensional axis gyroscopes 140,accelerometers 150, and magnetometers 160, the wearable virtual realitysystem 100 may calculate a “start” or “neutral” position of the user andthe VR environment from which further motion of the head of the user310—and by extension the HMD 200—are adjudged. Such calibration mayoccur at the beginning of operation, during a manual reset, or inresponse to an automatic determination by the wearable virtual realitysystem 100 that positional information has “drifted” or is no longercorrelating properly such that re-calibration is required. Suchdetermination may occur through execution of software stored in memory170 by processor 180.

Turning now to user 350 in FIG. 3, such user (which is the same user asuser 310 but simply having turned their head approximately 45 degrees)has turned their head such that their line-of-sight is no longerparallel along ray 330 as established during the aforementionedcalibration process. The new line-of-sight 340 ₁ reflects that theline-of-sight is now approximately 45 degrees (360) to the right of theoriginally established ray 330. By utilizing information generated byone or more of three-dimensional axis gyroscopes 140, accelerometers150, and magnetometers 160, the wearable virtual reality system 100 maycalculate how far the line-of-sight 340 ₁ has changed from ‘start’ or‘neutral’ position of the user and that was used to establish ray 330.

Like ray 330, angle 360 is illustrated for assisting in theunderstanding of the implementation of an environmental interrupt or“pause” feature whereby activities in the environment are interrupted orput on hold to allow for some other function, including but not limitedto menu navigation. But also like ray 330, angle 360 may be visuallyillustrated to the user in the virtual environment 320 as part of agraphical overlay. This information might be displayed as a geometricillustration showing the actual change in angle from center ray 330 ormerely as a numerical indicator of the number of degrees (e.g., 12degrees) of center 330 that the user has turned their head.

It should be noted that while an embodiment of the present inventionspecifically addresses an “interrupt” or “pause” functionality by way ofthe user turning their head in excess of a particular angle asillustrated in FIG. 3, other functionalities may be associated with thepositional change (e.g., save function, reset function, re-startfunction). In this regard, the interrupt or “pause” function isexemplary. Still further, an embodiment might implement different angleswith different functions. For example, “pause” might be implement after20 degrees off of center 330, whereas save might be implemented after 30degrees from center 330, and re-start after 45 degrees from center 330.Implementation of those functions may occur as soon as the degree changeis reached or after the user leaves their head in a particular positionchange for a predetermined period of time.

FIG. 4 illustrates a method 400 for implementing an interrupt in the VRenvironment. The method 400 of FIG. 4 may be embodied as executableinstructions in a non-transitory computer readable storage mediumincluding but not limited to a CD, DVD, or non-volatile memory such as ahard drive. Such methodology may be implemented by processor 180executing non-transitory computer readable instructions embodied inmemory 170. Processor 180 and software stored in memory 170 may utilizedata acquired from various other components of system 100 includingthree-dimensional axis gyroscopes 140, accelerometers 150, andmagnetometers 160. The steps identified in FIG. 4 (and the orderthereof) are exemplary and may include various alternatives,equivalents, or derivations thereof including but not limited to theorder of execution of the same.

In step 410, a calibration process may commence. The calibration mayoccur at start-up of wearable virtual reality system 100 or in responseto launching a particular application in the context of system 100. Auser may also request a manual calibration, or the system 100 mayrequire one due to positional drifts.

In response to the calibration process, information fromthree-dimensional axis gyroscopes 140, accelerometers 150, andmagnetometers 160 is received in step 420. This information will be usedto determine a neutral or “at rest” position from which all otherangular calculations will be contextually judged. This determination maycorrespond, for example, to ray 330 as discussed in the context of FIG.3. Measurements and calculations may take place on the X as well as theY axis. In this regard, “pause” or other functions may be introduced notonly by movements along the X-axis, but also along the Y-axis or even acombination of the two (e.g., a user raises their head to the right andbeyond a certain position).

In step 430, various controls may be set with respect to positional datagenerated in step 420. The neutral position of ray 330 may be confirmedas well as various functions that may be implemented if the positionaldata of HMD 200 indicates that the user has turned their line-of-sightbeyond a particular angle, which may include along a particular axis oraxes. In some instances, various functions may be implemented forincreasing angles of change. Time periods may also be implementedwhereby a user must change their line-of-sight along a particular axisbeyond a particular angle for a given period of time.

In step 440, tracking of HMD 200 commences using information generatedby the likes of three-dimensional axis gyroscopes 140, accelerometers150, and magnetometers 160. Throughout the tracking process, a continualcheck is made as to whether the position data of HMD 200 indicates thatit has exceeded one of the limitations or controls set in step 430. Forexample, and as shown in FIG. 3, a determination is made as to whetherthe user has moved their head and hence their line-of-sight 340 beyond aparticular control angle relative neutral ray 330. If the angle has notbeen exceeded (or not exceeded for a predefined period of time), thentracking continues at step 440, and checks relative to settings fromstep 430 continue to be made at step 450. If the user has, however,exceeded a positional setting along a particular axis for a particularperiod of time (or any other setting controlled at step 430), then thecorresponding functionality—such as a “pause”—may be implemented at step460.

FIG. 5 illustrates the use of non-field-of-view real estate to provideinformation ancillary to the VR environment. A user may be determined tohave turned their field of view beyond a neutral or center setting, suchas discussed in the context of FIG. 3. Because the user has paused theVR environment being displayed by HMD 200, the user may now attend toother activities in the real-estate areas that are not a direct part ofVR environment and that would typically be relegated to the “peripheralvision” areas of the line-of-sight of the user.

For example, this area might include various menus and controls relatedto the VR environment or the application currently executing to generatethe VR environment. It may further include data about the VR environmentsuch as status of activity taking place in the environment (e.g.,scores, health, inventory, etc.). The peripheral area real estate mightalso include status information concerning the system 100 or the HMD 200of the system 100. Advertisements might also be displayed in this area.Other applications might also execute in this area, such as video calls,messages, or other real-time communications. By using this space forsuch data and allowing the user to access the same during a pausedstate, the primary line-of-sight area in the VR environment can bebetter utilized.

The present invention may be implemented in an application that may beoperable using a variety of devices. Non-transitory computer-readablestorage media refer to any medium or media that participate in providinginstructions to a central processing unit (CPU) for execution. Suchmedia can take many forms, including, but not limited to, non-volatileand volatile media such as optical or magnetic disks and dynamic memory,respectively. Common forms of non-transitory computer-readable mediainclude, for example, a floppy disk, a flexible disk, a hard disk,magnetic tape, any other magnetic medium, a CD-ROM disk, digital videodisk (DVD), any other optical medium, RAM, PROM, EPROM, a FLASHEPROM,and any other memory chip or cartridge.

Various forms of transmission media may be involved in carrying one ormore sequences of one or more instructions to a CPU for execution. A buscarries the data to system RAM, from which a CPU retrieves and executesthe instructions. The instructions received by system RAM can optionallybe stored on a fixed disk either before or after execution by a CPU.Various forms of storage may likewise be implemented as well as thenecessary network interfaces and network topologies to implement thesame.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. The descriptions are not intended to limit the scope of theinvention to the particular forms set forth herein. Thus, the breadthand scope of a preferred embodiment should not be limited by any of theabove-described exemplary embodiments. It should be understood that theabove description is illustrative and not restrictive. To the contrary,the present descriptions are intended to cover such alternatives,modifications, and equivalents as may be included within the spirit andscope of the invention as defined by the appended claims and otherwiseappreciated by one of ordinary skill in the art. The scope of theinvention should, therefore, be determined not with reference to theabove description, but instead should be determined with reference tothe appended claims along with their full scope of equivalents.

What is claimed is:
 1. A method for executing a function within avirtual environment, the method comprising: storing information inmemory regarding at least one control setting that associates a functionwith a change in position of a head-mounted display, calibrating aneutral position for the head-mounted display, wherein the calibrationis performed using one or more sensors, monitoring positional dataassociated with the head-mounted display, wherein the monitoredpositional data is obtained via the one or more sensors, evaluating themonitored positional data of the head-mounted display against one ormore thresholds specified in the at least one control setting, andexecuting the function associated with the at least one control settingbased on the evaluated monitored positional data of the head-mounteddisplay.
 2. The method of claim 1, wherein the one or more sensors usedfor calibrating the neutral position includes three-dimensional axisgyroscopes, accelerometers and magnetometers.
 3. The method of claim 1,wherein the neutral position for the head-mounted display is withrespect to a vertical plane.
 4. The method of claim 1, wherein theneutral position for the head-mounted display is with respect to ahorizontal plane.
 5. The method of claim 1, wherein the neutral positionfor the head-mounted display is with respect to both a vertical planeand a horizontal plane.
 6. The method of claim 1, wherein the one ormore thresholds includes movement of the head-mounted display thatexceeds a pre-determined distance from the neutral position.
 7. Themethod of claim 6, wherein the one or more thresholds further includes apre-determined period of time that movement of the head-mounted displayexceeds a pre-determined distance from the neutral position.
 8. Themethod of claim 1, wherein the executed function includes interrupting avirtual environment associated with the head-mounted display by pausingthe environment.
 9. The method of claim 1 further comprising generatingcorresponding menu functionalities in a vision area for a user to viewwith the head-mounted display based on the executed function.
 10. Asystem for executing a function within a virtual environment, the systemcomprising: memory that stores information regarding at least onecontrol setting that associates a function with a change in position ofa head-mounted display; one or more sensors that: calibrates a neutralposition for the head-mounted display, wherein the calibration isperformed using one or more sensors, and monitors positional dataassociated with the head-mounted display, wherein the monitoredpositional data is obtained via the one or more sensors; and a processorthat executes instructions stored in memory to: evaluate the monitoredpositional data of the head-mounted display against one or morethresholds specified in the at least one control setting, and executethe function associated with the at least one control setting based onthe evaluated monitored positional data of the head-mounted display. 11.The system of claim 10, wherein the one or more sensors used forcalibrating the neutral position includes three-dimensional axisgyroscopes, accelerometers and magnetometers.
 12. The system of claim10, wherein the neutral position for the head-mounted display is withrespect to a vertical plane.
 13. The system of claim 10, wherein theneutral position for the head-mounted display is with respect to ahorizontal plane.
 14. The system of claim 10, wherein the neutralposition for the head-mounted display is with respect to both a verticalplane and a horizontal plane.
 15. The system of claim 10, wherein theone or more thresholds includes movement of the head-mounted displaythat exceeds a pre-determined distance from the neutral position. 16.The system of claim 15, wherein the one or more thresholds furtherincludes a pre-determined period of time that movement of thehead-mounted display exceeds a pre-determined distance from the neutralposition.
 17. The system of claim 10, wherein the executed functionincludes interrupting a virtual environment associated with thehead-mounted display by pausing the environment.
 18. The system of claim10, wherein the processor further executes instructions to generatecorresponding menu functionalities in a vision area for a user to viewwith the head-mounted display based on the executed function.
 19. Anon-transitory computer readable storage medium having embodied thereona program, the program being executable by a processor to perform amethod for executing a function within a virtual environment, the methodcomprising: storing information in memory regarding at least one controlsetting that associates a function with a change in position of ahead-mounted display, calibrating a neutral position for thehead-mounted display, wherein the calibration is performed using one ormore sensors, monitoring positional data associated with thehead-mounted display, wherein the monitored positional data is obtainedvia the one or more sensors, evaluating the monitored positional data ofthe head-mounted display against one or more thresholds specified in theat least one control setting, and executing the function associated withthe at least one control setting based on the evaluated monitoredpositional data of the head-mounted display.